AI-102 Azure AI Engineer
Associate — Full Course

☁️

What Are Azure AI Services?

• Azure AI Services = pre-built, cloud-hosted AI APIs from Microsoft — no model training required for most use cases
• Two resource types:
• Single-service: one endpoint, one key per service (e.g., Azure AI Vision only)
• Multi-service (Cognitive Services): one key, one endpoint for ALL services — simpler billing
• Always provisioned via: Azure Portal, Azure CLI, ARM templates, or Bicep
• Pricing tiers: Free (F0) — limited quota; Standard (S0) — pay-per-call
• Endpoint format: https://<resource-name>.cognitiveservices.azure.com/

🔑

Authentication Methods

💻

Python Authentication Examples

# ── Option 1: Key-Based Authentication ──
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from msrest.authentication import CognitiveServicesCredentials

client = ComputerVisionClient(
    endpoint="https://my-vision.cognitiveservices.azure.com/",
    credentials=CognitiveServicesCredentials("YOUR_KEY")
)

# ── Option 2: Managed Identity (PRODUCTION BEST PRACTICE) ──
from azure.identity import DefaultAzureCredential
from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import TokenCredential

credential = DefaultAzureCredential()  # auto-picks env/MI/CLI
client = TextAnalyticsClient(
    endpoint="https://my-language.cognitiveservices.azure.com/",
    credential=credential
)

# ── Key Vault integration (retrieve key at runtime) ──
from azure.keyvault.secrets import SecretClient

kv_client = SecretClient(
    vault_url="https://my-vault.vault.azure.net/",
    credential=DefaultAzureCredential()
)
api_key = kv_client.get_secret("vision-api-key").value

👥

RBAC Roles for AI Services

🔒

Network Isolation Options

• Default behavior: AI Services accept traffic from any IP (public endpoint)
• IP Firewall rules: restrict to specific public IP ranges — still uses public endpoint
• VNet Service Endpoints: route traffic through Azure backbone; endpoint remains public but only accessible from specified subnets
• Private Endpoint (most secure): assigns a private IP inside your VNet — no public exposure
• DNS: privatelink.cognitiveservices.azure.com resolves to private IP
• After creating private endpoint, disable public access entirely
• Requires Azure Private DNS Zone for name resolution

🗝️

Customer-Managed Keys (CMK)

• Default: Microsoft manages encryption keys (platform-managed) — you cannot revoke
• CMK: your organization's key stored in Azure Key Vault encrypts data-at-rest
• Revocation: disable or delete the key in Key Vault → service data becomes immediately inaccessible
• Requirements: Standard pricing tier + Azure Key Vault with soft-delete + purge protection enabled
• Setup: Key Vault → Grant AI Service identity access → Configure CMK in AI resource settings

📈

Azure Monitor Integration

• Metrics (built-in, no config): TotalCalls, TotalErrors, Latency, SuccessRate
• Diagnostic Logs: enable via Portal → Monitoring → Diagnostic settings
• Send to: Log Analytics Workspace (KQL queries), Storage Account (archive), Event Hub (stream)
• Alerts: trigger on metric thresholds — e.g., "TotalErrors > 100 in 5 min"
• Application Insights: distributed tracing for SDK-based apps — trace IDs, spans, custom telemetry

🔍

Useful KQL Queries

// All AI Service diagnostic logs
AzureDiagnostics
| where ResourceType == "COGNITIVESERVICES"
| project TimeGenerated, OperationName, ResultType, DurationMs

// Error rate over time
AzureMetrics
| where MetricName == "TotalErrors"
| summarize sum(Total) by bin(TimeGenerated, 5m)
| render timechart

// Latency P95
AzureMetrics
| where MetricName == "Latency"
| summarize percentile(Average, 95) by bin(TimeGenerated, 1h)

⚖️

Microsoft's 6 Responsible AI Principles

🛡️

Azure AI Content Safety

• Dedicated service for detecting harmful content in text AND images
• 4 harm categories: Hate, Sexual, Violence, Self-harm
• Severity levels: 0 (safe) to 6 (very harmful)
• Prompt Shield: detect jailbreak attempts and indirect prompt injection
• Groundedness detection: check if AI response is supported by retrieved context (RAG)
• Protected material detection: identify copyrighted text/code in model output

👁️

Image Analysis Features

💻

Code Sample — Analyze Image

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.ai.vision.imageanalysis.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential

client = ImageAnalysisClient(
    endpoint="https://my-vision.cognitiveservices.azure.com/",
    credential=AzureKeyCredential("YOUR_KEY")
)

result = client.analyze_from_url(
    image_url="https://aka.ms/azsdk/image-analysis/sample.jpg",
    visual_features=[
        VisualFeatures.CAPTION,        # "a dog running on grass"
        VisualFeatures.TAGS,           # dog, outdoor, green, fun
        VisualFeatures.READ,           # OCR text extraction
        VisualFeatures.DENSE_CAPTIONS, # per-region captions
        VisualFeatures.OBJECTS,        # bounding boxes
    ],
    gender_neutral_caption=True,
)

# Results
print(f"Caption: {result.caption.text} ({result.caption.confidence:.2f})")
for tag in result.tags.list:
    print(f"  Tag: {tag.name} ({tag.confidence:.2f})")
for block in result.read.blocks:
    for line in block.lines:
        print(f"  OCR: {line.text}")

🎯

Custom Vision Overview

• Two project types: Classification and Object Detection
• Classification: assign one (multi-class) or many (multi-label) categories per image
• Object Detection: locate and label objects with bounding boxes
• Minimum images: 15 per tag for classification; 15 labelled regions for detection
• Training modes: Quick Training (fast, less accurate) vs Advanced Training (slower, better)
• Domain types: determine export options
• General (Compact): exportable to CoreML, TensorFlow, ONNX, Docker (edge/offline)
• General (S1/A2): cloud API only — cannot be exported
• Portal: customvision.ai

📤

Export Formats for Edge Deployment

🧑

Face API Operations

💻

Face Identification Flow

# 1. Create PersonGroup
face_client.person_group.create(person_group_id="employees", name="Employees")

# 2. Add persons and their face images
person = face_client.person_group_person.create("employees", name="Alice")
face_client.person_group_person.add_face_from_url(
    "employees", person.person_id, "https://example.com/alice.jpg"
)

# 3. Train the PersonGroup
face_client.person_group.train("employees")

# 4. Detect faces in a new photo
detected = face_client.face.detect_with_url("https://example.com/newphoto.jpg")
face_ids = [f.face_id for f in detected]

# 5. Identify against the group
results = face_client.face.identify(face_ids, person_group_id="employees")
for r in results:
    if r.candidates:
        print(f"Identified: {r.candidates[0].person_id} ({r.candidates[0].confidence:.2f})")

🎬

Video Indexer Insights

• Account types: Trial (limited uploads, free) vs. Paid (connected to Azure Media Services)
• Indexing presets: Basic (audio only), Standard (audio+video), Advanced (+ deep analysis)
• Widgets: embed Player Widget + Insights Widget in any web app via iframe
• Access: videoindexer.ai portal or REST API with access token

📝

Pre-built Language Features

💻

Code: Sentiment + Opinion Mining

from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

client = TextAnalyticsClient(
    endpoint="https://my-language.cognitiveservices.azure.com/",
    credential=AzureKeyCredential("KEY")
)

docs = ["Azure is powerful but the pricing is confusing."]

# Opinion mining reveals aspect-level sentiment
results = client.analyze_sentiment(docs, show_opinion_mining=True)
for doc in results:
    print(f"Overall: {doc.sentiment}")
    for sentence in doc.sentences:
        for opinion in sentence.mined_opinions:
            target = opinion.target
            print(f"  Target: {target.text} → {target.sentiment}")
            for assessment in opinion.assessments:
                print(f"    Assessment: {assessment.text} ({assessment.sentiment})")

# Output:
# Overall: mixed
# Target: Azure → positive  Assessment: powerful (positive)
# Target: pricing → negative  Assessment: confusing (negative)

❓

Question Answering

• Successor to QnA Maker — part of Azure AI Language
• Create a knowledge base from: FAQ URLs, PDF/Word docs, or manual QA pairs
• Custom QA — train on your content; Prebuilt QA — no project needed
• Confidence score threshold: default 0, recommend setting ≥0.5 in production
• Multi-turn conversations: define follow-up prompts for clarification flows
• Chitchat: add personality (professional, friendly, witty) for small talk
• Active learning: uses low-confidence user queries to suggest new QA pairs

🗣️

Conversational Language Understanding (CLU)

• Successor to LUIS — same concept, improved model
• Intents = what the user wants (e.g., BookFlight, CheckWeather)
• Entities = key info extracted (e.g., City, Date, FlightClass)
• Minimum utterances per intent: 5 recommended (15–30 for production)
• Orchestration Workflow — meta-project that routes to CLU, QA, or LUIS based on input
• Deploy to Language endpoint → call via REST or SDK with analyze_conversation()

🎤

Speech-to-Text (STT)

• Real-time recognition: live microphone or streaming audio
• Batch transcription: async processing of large audio files (Azure Blob) — use for 500+ hours
• Custom Speech: upload audio + transcripts to improve accuracy for domain-specific vocabulary
• Speaker Diarization: labels utterances by speaker — "who said what" — enabled via diarization_config
• Keyword Recognition: detect specific wake words locally without cloud calls

🔊

Text-to-Speech (TTS) & SSML

• 400+ voices, 140+ languages. Neural voices = highly natural speech
• Custom Neural Voice: clone a voice from 300+ recordings — Limited Access
• SSML (Speech Synthesis Markup Language) — control output precisely:

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">
  <voice name="en-US-JennyNeural">
    <prosody rate="slow" pitch="+10%">
      Welcome to the Azure AI course.
    </prosody>
    <break time="2s"/>
    <emphasis level="strong">This is important!</emphasis>
  </voice>
</speak>

🌍

Translator Key Facts

• Neural Machine Translation for 100+ languages
• Global endpoint: api.cognitive.microsofttranslator.com — NOT regional!
• Required headers: Ocp-Apim-Subscription-Key AND Ocp-Apim-Subscription-Region
• Operations: Translate, Transliterate, Detect, Dictionary Lookup, Dictionary Examples, BreakSentence
• Custom Translator: upload parallel bilingual documents → domain-specific model
• Document Translation: async translation of entire files (PDF, Word, PowerPoint) maintaining formatting

import requests, uuid

url = "https://api.cognitive.microsofttranslator.com/translate"
params = {"api-version": "3.0", "from": "en", "to": ["fr", "ar"]}
headers = {
    "Ocp-Apim-Subscription-Key": "YOUR_KEY",
    "Ocp-Apim-Subscription-Region": "eastus",  # REQUIRED!
    "Content-Type": "application/json",
    "X-ClientTraceId": str(uuid.uuid4())
}
body = [{"text": "Azure AI is amazing!"}]

response = requests.post(url, params=params, headers=headers, json=body)
print(response.json())
# [{"translations": [{"text": "Azure AI est incroyable!", "to": "fr"}, ...]}]

🔎

Azure AI Search Architecture

🔬

Query Types

⚙️

Built-in Skills

💾

Knowledge Store

• Persists AI enrichments from indexing to Azure Storage (outside the search index)
• Three projection types:
• Tables → Azure Table Storage (rows per document/chunk)
• Objects → Azure Blob as JSON files
• Files → Azure Blob as normalized images
• Used for: Power BI analysis, downstream ML pipelines, audit trails
• generatedKeyName defines the RowKey field in table projections

📄

Pre-built Models

🎓

Custom Models

• Custom Template: fixed-layout forms (consistent field positions) — minimum 5 labelled samples
• Custom Neural: variable-layout forms — requires ~15+ labelled samples, handles diverse layouts
• Composed Model: bundle multiple custom models under one model ID — service picks the best match
• Label using the Document Intelligence Studio: documentintelligence.ai.azure.com
• selectionMarks output = checkboxes and radio buttons with selected/unselected state

🤖

Azure OpenAI Fundamentals

💻

Chat Completions API

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://my-openai.openai.azure.com/",
    api_key="YOUR_API_KEY",
    api_version="2024-02-01"
)

response = client.chat.completions.create(
    model="gpt4o-prod",  # YOUR deployment name, not "gpt-4o"!
    messages=[
        {"role": "system",  "content": "You are a helpful Azure AI expert."},
        {"role": "user",    "content": "Explain Azure AI Search in 3 points."}
    ],
    temperature=0.3,    # 0=deterministic, 1=creative
    max_tokens=500,
    top_p=0.9,
)

print(response.choices[0].message.content)
print(f"Tokens used: {response.usage.total_tokens}")
print(f"Finish reason: {response.choices[0].finish_reason}")
# finish_reason: "stop" | "length" | "tool_calls" | "content_filter"

✏️

Prompting Techniques

🎛️

Parameter Reference

🗃️

RAG Pipeline on Azure

• Step 1 — Ingest: chunk documents into ~512 token segments → generate embeddings via text-embedding-3-small
• Step 2 — Store: vector index in Azure AI Search (HNSW) or Cosmos DB for MongoDB vCore
• Step 3 — Retrieve: embed user query → ANN search → top-K chunks
• Step 4 — Generate: inject chunks into prompt → GPT-4o generates grounded answer
• System prompt: "Only answer from the provided context. Say 'I don't know' if not found."

# 1. Generate embedding for the user query
embedding_response = client.embeddings.create(
    model="text-embedding-3-small",  # deployment name
    input=user_query
)
query_vector = embedding_response.data[0].embedding

# 2. Vector search in AI Search
from azure.search.documents.models import VectorizedQuery
results = search_client.search(
    search_text=None,
    vector_queries=[VectorizedQuery(
        vector=query_vector, k_nearest_neighbors=5,
        fields="content_vector"
    )]
)
chunks = [r["content"] for r in results]

# 3. Generate grounded answer
context = "\n\n".join(chunks)
response = client.chat.completions.create(
    model="gpt4o-prod",
    messages=[
        {"role": "system", "content": f"Answer ONLY from:\n\n{context}"},
        {"role": "user",   "content": user_query}
    ]
)

🎨

DALL-E 3 Image Generation

• Resolutions: 1024×1024, 1024×1792 (portrait), 1792×1024 (landscape)
• Style: vivid (hyperrealistic) or natural (softer, less saturated)
• Quality: standard or hd (more detail, higher cost)
• Response includes revised_prompt — DALL-E may rewrite for safety/quality
• Image URL expires after 1 hour — download and store if needed

result = client.images.generate(
    model="dall-e-3",   # your deployment name
    prompt="A futuristic Azure data center glowing blue in space",
    n=1,
    size="1792x1024",
    quality="hd",
    style="vivid"
)
print(result.data[0].url)            # temporary URL (1 hr expiry)
print(result.data[0].revised_prompt)  # what DALL-E actually used

🛡️

Azure AI Content Safety Features